A Longitudinal Study of Topic Classification on Twitter

Twitter represents a massively distributed information source over a kaleidoscope of topics ranging from social and political events to entertainment and sports news. While recent work has suggested that variations on standard classifiers can be effectively trained as topical filters (Lin, Snow, and Morgan 2011; Yang et al. 2014; Magdy and Elsayed 2014), there remain many open questions about the efficacy of such classification-based filtering approaches. For example, over a year or more after training, how well do such classifiers generalize to future novel topical content, and are such results stable across a range of topics? Furthermore, what features and feature classes are most critical for longterm classifier performance? To answer these questions, we collected a corpus of over 800 million English Tweets via the Twitter streaming API during 2013 and 2014 and learned topic classifiers for 10 diverse themes ranging from social issues to celebrity deaths to the “Iran nuclear deal”. The results of this long-term study of topic classifier performance provide a number of important insights, among them that (1) such classifiers can indeed generalize to novel topical content with high precision over a year or more after training and (2) simple terms and locations are the most informative feature classes (despite training on classes labeled via hashtags). 1 Learning Topical Social Sensors Our objective is to evaluate binary classifiers that can label a previously unseen tweet as topical (or not). Following the approach of (Lin, Snow, and Morgan 2011), for a topic t, we leverage a (small) set of user-curated topical hashtagsH to efficiently provide a large number of supervised topic labels for training. As standard for machine learning methods, we divide our training data into train and validation sets — the latter for hyperparameter tuning to control overfitting and ensure generalization to unseen data. As a critical insight for topical generalization where we view correct classification of tweets with previously unseen topical hashtags as a proxy for topical generalization, we do not simply split our data temporally into train, validation, and test sets and label both with all hashtags inH. Instead, we splitH into three disjoint sets H train, H t val, and H t test according to two time stamps t split and t test split for topic t and the first usage time Copyright c © 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. #Unique Features From Hashtag Mention Location Term 95,547,198 11,183,410 411,341,569 58,601 20,234,728 Feature Usage in #Tweets Feature Max Avg Median Most frequent From 10,196 8.67 2 running status Hashtag 1,653,159 13.91 1 #retweet Mention 6,291 1.26 1 tweet all time Location 10,848,224 9,562.34 130 london Term 241,896,559 492.37 1 rt Feature Usage by #Users Hashtag 592,363 10.08 1 #retweet Mention 26,293 5.44 1 dimensionist Location 739,120 641.5 2 london Term 1,799,385 6,616.65 1 rt Feature Using #Hashtags From 18,167 2 0 daily astrodata Location 2,440,969 1,837.79 21 uk Table 1: Feature Statistics of our 829, 026, 458 tweet corpus. stamp htime∗ of each hashtag h ∈ H. In short, all hashtags h ∈ H with htime∗ < t split are used to generate positive labels in the training data, those with htime∗ ≥ t split are used for positive labels in the test data and the remainder are used for positive labels in the validation data. The key point to observe is that we not only partition the train, validation, and test data temporally, but we also divide the hashtag class labels temporally and label each data partition with an entirely disjoint set of topical hashtags. The purpose behind this training and validation data split and labeling is to ensure that learning hyperparameters are tuned so as to prevent overfitting and maximize generalization to unseen topical content (i.e., new hashtags). We remark that a classifier that simply memorizes training hashtags will fail to correctly classify the validation data except in cases where a tweet contains both a training and validation hashtag.